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Abstract 

This study was conducted to determine whether states with 
performance budgeting and funding (PBF) programs had 
improved institutional performance of higher education over the 
five years (1997 through 2001) considered in this study. First 
Time in College (FTIC) graduation rate was used as the measure 
of institutional performance. In this study, the unit of analysis is 
institution level and the study population is all public four-or-more- 
year institutions in the United States. To test PBF program 
effectiveness. Hierarchical Linear Modeling (HLM) growth analysis 
was applied. According to the HLM analysis, the growth of 
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graduation rates in states with PBF programs was not greater 
than in states without PBF programs. The lack of growth in 
institutional graduation rates, however, does not mean that PBF 
programs failed to achieve their goals. Policy-makers are advised 
to sustain PBF programs long enough until such programs bear 
their fruits or are proven ineffective. 

Introduction 

The purpose of this study was to examine the effects of performance budgeting 
and funding (PBF) programs on the performance of public higher education 
institutions in the United States using longitudinal performance data. As of 2001, 
thirty-six states in the U.S. utilized either performance budgeting or performance 
funding programs, or both, in institutions of higher education (Burke & 
Minassians, 2001). This trend began during the 1990s, as state policy-makers 
sought to enhance accountability in higher education, find more effective budget 
allocation methods, and in some cases, trim budgets. Although performance 
budgeting (PB) and performance funding (PF) are different in focus, both create 
a link, to different degrees, between budget allocation and institutional 
performance. In addition, both types of programs are based on the underlying 
premise that higher education institutions are motivated to improve their 
performance when performance is linked to budget allocation. 

The trend toward increased use of performance budgeting and funding 
programs is not limited to the U.S. The United Kingdom led other European 
countries in initiating a market-driven higher education accountability program, 
first introduced by the Thatcher government. Influenced by trends in the U.S and 
Europe, the Canadian provinces of Alberta and Ontario have also adopted 
performance-based funding mechanisms to assess university efficiency and 
quality (Barnetson 1999). In Australia, 40 public research universities and two 
private institutions are subject to an accountability framework that links their 
funding to performance requirements (Atkinson-Grosjean & Grosjean, 2000). In 
spite of vigorous protests from students and faculty. New Zealand established a 
new accountability system in 1998. In the early 1990s, South Korea adopted a 
performance-based funding program consisting of nine independent sub- 
programs, each with a particular purpose, to which funding was tied (Kim, 

2001 ). 

Current Status Of Pbf Programs 

Major Stakeholders: State Politics and Business Invoivement 

The major stakeholders in higher education reform of the 1990s have been state 
legislatures, state chancellors and state agencies, governors, and higher 
education institutions (Blackwell & Ciston, 1998). Among these stakeholders, 
governors and state legislatures play the primary role in the adoption of PBF 
programs (Burke & Minassians, 2001). In contrast, a relatively small number of 
initiatives (in 24 states) have been conducted within state agencies or university 
systems themselves. 
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A more complete understanding of the widespread adoption of PBF programs 
since the 1990s must encompass more than the vested interests of state 
legislatures and governors. In the 1990s, higher education reform has been 
guided by economic values — competition, productivity, and efficiency — more 
than ever before. This shift in priorities might be explained in part by the growing 
interest of business leaders in state higher education policy-making (Usdan, 
2002; Zernike, 2002; Zumeta, 2000). 

Business interests have changed the picture of higher education. As Levin 
(2001) argues, higher education is becoming like a “globally competitive 
business” (p.237). In some cases, business leaders are actively involved in PBF 
program development. For example, in South Carolina, which links 100% of 
higher education funding to institutional performance, two of the 12 members of 
the Higher Education Joint Legislative Study Committee represented the 
business community. [Not one of the 12 members of the committee came from 
higher education (China, 1998)]. 

However, other major stakeholders in higher education — institutional 
administrators and faculty — have always been more concerned about 
institutional improvement than accountability, unlike politicians who emphasize 
external accountability through the implementation of PBF programs (Burke, 
1998). Because of these perceptual and value differences, tensions almost 
invariably exist between both groups around the issue of PBF programs. 

Type of Program: Performance Budgeting or Funding? 

Performance budgeting and funding are similar in that both programs link 
institutional performance with budget allocation, but the methods differ in the 
way each ties institutional performance to state funding. While performance 
funding (PF) programs link budget to institutional performance in a direct, 
automatic, and formulaic manner, the link in performance budgeting (PB) 
programs is loose, indirect, and uncertain (Burke et al, 2000). 

Because of the loose link between performance and budget allocation, PB 
programs may be less effective in enhancing institutional performance. 
Notwithstanding this limitation, state policy-makers typically prefer performance 
budgeting to performance funding because of the flexibility in implementation 
(Burke, et al., 2000). 

Although more states adopted PB programs than PF programs, recent trends 
indicate, “both programs are borrowing elements from the other approach to 
gain its benefits while evading their own problems” (Burke et al, 2000, p.3). 
Further, in order to minimize the limitations and maximize the strengths of each 
program, an increasing number of states have adopted both programs at the 
same time (Burke et al, 2000). 

Commonly-Used Indicators for measuring institutional 
performance 
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Ruppert (1995) conducted an in-depth case study on performance indicators 
(Pis) and performance reporting in ten states that have been among the leaders 
in designing and using Pis in higher education. (Note 1 ) He found that most 
states share a common core of performance indicators. Graduation outcome 
data, for instance, is one measure that responds to policy concerns — rising 
college costs and the economic return to the state of college-educated citizens. 

A survey by the State Higher Education Executive Officers (SHEEO) conducted 
between 1996 and 1997 identifies some indicators used frequently in the U.S. at 
that time; graduation rate (32 states), transfer rate (25 states), faculty 
workload/productivity (24 states), follow-up satisfaction (23 states), and 
externally sponsored research funds (23 states) (Christal, 1998). Another study 
(Burke, 1998) of 10 states conducted in 1997 found less commonality among 
Pis in different PBF programs. Among the states surveyed, the following 
performance indicators were in use; graduation and retention rates (10 states), 
two-to-four-year transfers (6 states), faculty workload (5 states), institutional 
choice (5 states), graduation credits and time to degree (4 states), licensure test 
score (4 states), transfer graduation rates (4 states), workforce training and 
development (4 states), and external research fund ratio (3 states). Overall, the 
common performance indicators across these studies were graduation rate, 
retention rate, transfer rate, faculty workload, and sponsored research funds. 

Previous Evaluations of the Impacts of Pbf Programs 

Most evaluative studies of the impacts of PBF programs focus on a specific 
state or set of institutions within a state. As many studies assert, the 
effectiveness of PBF programs is a critical concern of policy-makers. 
Nevertheless, few academic evaluation studies of PBF program impacts have 
been conducted, save the occasional evaluation project or dissertation. State- 
level evaluations for budget allocation purposes mainly use quantitative data on 
specific performance indicators. Florida and South Carolina, for example, 
annually evaluate each institution’s performance in order to guide budget 
allocation. 

In a book on PBF programs, Bogue (2002) explores the impact and 
effectiveness of the PBF program in Tennessee. From the actual 1994-1998 
performance data, he found that the universities have consistently scored above 
the national norm on the College BASE. One interesting outcome of the 
Tennessee PBF program is the clear increase in the proportion of academic 
program accreditations, which improved from 65 percent to 100 percent (Bogue, 
2002). He found, however, that institutions generally failed to significantly raise 
persistence-to-graduation rates and overall job-placement rates over the same 
period. 

Florida’s Office of Program Policy Analysis and Government Accountability 
(OPPAGA) (2001) evaluated the performance budgeting program of the State 
University System in Florida as part of an evaluation of performance budgeting 
programs at each state agency. In the evaluation, OPPAGA reported that 
graduation and retention rates have increased since the state adopted a 
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performance budgeting program, and externally-funded financial support for the 
research program has significantly increased as well. Additional confirmation of 
these findings exists in Florida’s relatively high score on completion indicators 
versus other evaluation indicators in the ‘Measuring Up 2000’ study. OPPAGA 
claims this result occurred because the Florida Legislature decided to make 
raising the completion or graduation rates a priority in its indicator system. 

The South Carolina Commission on Higher Education (CHE) (2002) has been 
studying the impacts of performance funding program in South Carolina as a 
three-year project for the Fund for Improvement of Postsecondary Education 
(FIPSE) grant. In this evaluation, the South Carolina CHE used diverse methods 
(journal analysis, surveys, interviews, and historical data) to review its PF 
program. According to preliminary findings, institutional policies have changed 
to accommodate the PF program and some outcomes are changing as well. 
Interestingly, the impacts of the PF program seem to vary depending on which 
evaluation method is used in the evaluation process. For example, respondents 
to the survey answered that the PF program has had no impact on institutional 
quality or efficiency. Yet historical data on institutional performance shows that 
performance on SAT scores, minority enrollment, and externally funded 
research have all increased, and that graduation rates are slowly rising as well. 
These results suggest that the actual performance of South Carolina higher 
education institutions has been increasing since South Carolina adopted the PF 
program. 

In a review of performance data from Missouri higher education institutions, 
Marcus et al. (2000) found that student assessment scores, licensure exam 
pass rates, first time in college (FTIC) graduation rates, minority graduation 
rates, and student job placement rates have all increased since Missouri 
adopted a performance funding program. While the causal relationship between 
PF and increased performance was not conclusive, the PF program was 
determined to be “clearly responsible for the identification of priorities for 
funding, for the establishment of assessment measures, and for helping 
institutions to accept that part of their state allocation is linked to 
results” (p.215). 

The studies mentioned above indicate that PBF programs have had a positive 
impact on institutional performance. However, these results are based on limited 
time periods, and the interpretation of the performance data can differ from one 
researcher to another. For example, Dallet, et al. (2002) analyzed the same 
performance data used in an earlier study by OPPAGA (2001) to discuss the 
impact of Florida’s PBF program. While OPPAGA reported that the PBF 
program had had a positive impact on Florida’s higher education institutions, the 
conclusion of Dallet and his colleagues was quite different. They argued that, 
although the graduation rate and retention rate in Florida’s higher education 
institutions increased somewhat after the inception of the PBF program in 1994, 
this growth actually occurred at a level similar to that of the 1990-91 academic 
year, when no such policy was introduced. In Colorado, which adopted a PF 
program from 1994/95 to 1996/97, the effect of performance level on budget 
reward was found in a regression analysis to be lower than expected (Bridges, 
1999). Instead, there was a higher correlation between institution size and 
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performance funding than between performance funding and performance level. 

Research Questions 

The overarching purpose of this study is to explore the impacts that PBF 
programs have had on institutional performance. Thus, there are two basic 
research questions addressed in this study: 

1 . Has institutional performance in states with PBF programs (PBF states) 
improved more than in states without PBF programs (non-PBF states)? 

2. Has institutional performance in states with both programs (PB and PF) 
improved more than in states with only a PB or PF program? 

Research Design And Method 

Unit of Analysis and Population 

The unit of analysis for this study is higher education institution, rather than the 
state, because the target of PBF programs is institution-level change, and state- 
level performance is simply the aggregation of each state’s institution-level 
performance. The study will be limited to public higher education institutions 
because private institutions are not the main objects of state policy. Further, 
although some states have the same PBF programs for public four-year 
institutions and community colleges, many states (e.g., Florida, California, New 
York) have different PBF programs for public four-year institutions and 
community college systems (Burke, 1998). In addition, as Hoyt (2001) 
addresses, the missions of community colleges are different from those of four- 
year colleges. Therefore, for this study the object of interest is limited to public 
four-year institutions, and excludes public community colleges. Also, because 
PBF programs focus mainly on undergraduate education, graduate-only 
institutions will be excluded from the analysis. 

Thus, the target population of this study is all public four-year higher education 
institutions in the United States. Based on the criteria above, 456 institutions 
were selected for the study population. (Note 2) The information on the annual 
PBF program status of each state is available from surveys conducted each 
year from 1997 through 2001 (Burke & Minassians, 2001; Burke etal., 2000; 
Burke & Modarresi, 1999; Burke & Serban; 1998; Burke & Serban, 1997). 

Dependent Variable 

The dependent variable used in this study is the First Time in College (FTIC) 
graduation rate of each institution during the years 1997 through 2001. This 
dependent variable was selected based on three criteria: 1) it is the most 
commonly-used performance indicator nationwide; 2) nationwide data are 
readily available; (Note 3) and 3) there is internal validity in using graduation 
rate as a measure of performance in higher education. The term is also in 
accordance with the Student Right-To-Know Act (SRTK), which mandated 
institutions to report the graduation rate of only “first-time” and “full-time” 
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students who spent “up to 150 percent of normal time to complete their 
degrees” (9PL 101-542; Federal Register, 1992). 

In conducting a broad policy studies such as this, one must consider the 
question of whether a given policy is accomplishing its goals (Affholter, 1994). In 
PBF programs, the attainment of program goals is measured using performance 
indicators that are chosen by state legislatures, coordinating boards, or in some 
cases, higher education institutions themselves. According to an analysis of 
performance indicators nationwide (Christal, 1998), graduation rate is the most 
commonly-used indicator in U.S. higher education institutions with PBF 
programs. Of the 33 states with PBF programs in 1997, 32 states used 
graduation rate as one of their performance indicators. 

In addition, when an inappropriate or weak indicator is chosen as a dependent 
variable, it can invalidate the results of a policy study. Each institution admits 
applicants who are deemed qualified to study at the institution. These students 
study approximately four years to satisfy graduation requirements that are set 
forth by the department, college, university, or state. If many students do not 
graduate in a timely manner, these students have either not yet satisfied the 
graduation requirements, have chosen to leave, or have transferred to another 
institution. Any of these cases poses a problem for the institution, which, due to 
the very nature of its mission, seeks to maintain a high graduation rate. As a 
result, low graduation rates may imply that the institutions are falling short in 
performing some of their functions. Therefore, graduation rate is a persuasive 
and valid indicator to represent how well the institution performs in at least one 
of its many goals. (Note 4) 

FTIC graduation rate is thus calculated using the completions within 150% of 
normal time to degree divided by the total number in a particular year’s cohort. 
The Integrated Postsecondary Education Data System (IPEDS) provides total 
cohort information, completion within 150% of normal time to degree, and the 
final FTIC graduation rate. Therefore, for the purpose of this study, the final 
graduation rate provided by the IPEDS will be used without any adjustment. 

Independent Variables 

Independent variables in the statistical model will be of two types; program (or 
treatment) variables, and control variables. The program variables are the PBF 
program- related factors. The control variables include factors other than 
program-related factors that influence graduation rate. PBF program variables 
include; 

1 . PBF program adoption 

a. PBF statesare states that have adopted PBF programs in at least 
three of the five years 1997-2001. 

b. Non-PBF statesare states that either did not adopt a PBF program or 
had adopted one for only one year during 1 997-2001 . (Note 5) 

2. PBF type 

a. PB & PF states are states that have concurrent performance funding 
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and performance budgeting programs, 
b. PF or PF statesare states that only have one type of program; PB or 
PF. 

Control variables are used to control for other factors that affect FTIC graduation 
rate. Based on the literature, several variables were chosen to control for 
exogenous influences on the graduation rate in order to accurately capture the 
effects of PBF programs. These control variables are divided into institution- 
level control variables and state-level control variables. 

Institution-level control variables are variables that have been shown to affect 
graduation rate at the institution level. The average tuition has been associated 
with graduation rates (Heller, 1997; Hsing & Chang, 1996; Dayhoff, 1991). The 
availability of financial aid (grants, loans, work study, etc.) has a positive impact 
on enrollment (John et al., 2001 ; Braunstein et al, 1 999; Sheridan et al., 1 994; 
Cabrera et al., 1992). The opportunity to live in a campus residence hall, 
especially during the freshman year, has also been shown to influence retention 
rates and time to completion (Astin, 1997; Sheridan et al., 1994). 

a. Average Tuition is defined as [(instate tuition + out of state tuition)/2]. 

b. Student Dormitory Ratio is defined as (space availability in dormitory 
facilities)/(full-time students). 

c. Financial aid student ratio is the percentage of full-time, first-time degree 
and certification seeking students who receive any financial aid (grants, 
loans, assistantships, fellowships, tuition wavier, tuition discounts, 
veterans benefits, employer aids and other monies). 

State-level control variables are state level characteristics which have been 
shown to influence graduation rate, and which must be included in the causal 
model in order to capture accurately the effects of PBF programs. Astin (1997), 
Schmitz (1993), and Kahn and Nauta (2001) found that college preparation 
factors (mean entrance test score on ACT or SAT, and freshmen’s high school 
grades) are positively associated with graduation rate. Also, unemployment rate 
has been shown to be negatively associated with graduation rate (Heller, 1999; 
Hsing & Chang, 1996). Statewide family ability to pay college costs is the 
variable that reflects financial support from the family to the students 
(Braunstein et al., 1999; John, 1993) as well as the economic conditions of each 
state. 

a. College preparation is defined as “Overall score” of college preparation 
(Note 6) as described in “Measuring Up 2000” (NCPPHE, 2001). 

b. Family ability to pay college costs is defined as the student’s family’s 
ability to pay at public four-year colleges, as described in “Measuring Up 
2000” (NCPPHE, 2001). 

c. Unemployment rate is defined as the number of unemployed as a percent 
of the labor force (U.S. Department of Labor, 2002). 

Data Sources 
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Study data are available directly or indirectly from diverse sources. Data on PBF 
program variables have been collected in annual surveys conducted by Burke 
and associates in the years 1997 through 2001 (Burke & Minassians, 2001; 
Burke et al., 2000; Burke & Modarresi, 1999; Burke & Serban, 1998; Burke & 
Serban, 1997). Nationwide data on state-level control variables were included in 
the nationwide performance evaluation study, “Measuring Up 2000” (NCPPHE, 
2001). The data on unemployment rate are available from the U.S. Department 
of Labor (U.S. Department of Labor, 2002). All the other control variables, and 
the dependent variable, graduation rate, are available from the IPEDS 
database. 

Method and Analysis Procedures 

Identifying the data structure of a dependent variable and independent variable 
is the first step in choosing the most appropriate statistical method. In this study, 
the dependent variable, FTIC graduation rate, is a continuous variable, while the 
independent variables consist of continuous variables and dichotomous 
variables. The data to be analyzed have two important characteristics. First, the 
data related to the dependent variable are longitudinal in nature, and will allow 
for an analysis of changes in the FTIC graduation rate in the years 1997 through 
2001 . Second, as discussed in a previous section, the data related to the 
independent variables have two hierarchical structures: institution-level 
variables and state-level variables. 

Considering the data structure and research questions, recently developed 
Hierarchical Linear Modeling (HLM) growth model will be applied. HLM growth 
analysis enables to describe changes over time in longitudinal data (unlike a 
pre-test and post-test design), and analyzes program impacts within a particular 
period of time specified by the researcher (unlike time-series analysis). What is 
more, when the data structure is hierarchical, the HLM growth model can better 
estimate the contribution of variables at each level (Arnold, 1992; Bryk & 
Raudenbush, 1992). Thus, under a hierarchical data structure, the HLM growth 
model is more relevant and useful than other methods such as pre-and post-test 
design or interrupted time series design. 

One strength of the HLM growth model is that HLM considers each different 
variable at each different level, and each different level is formally represented 
by its own sub-model — institution-level and state-level in this study. These sub- 
models express relationships between variables at each level, and specify how 
variables at one level influence relations occurring at another (Bryk & 
Raudenbush, 1992). When the change in FTIC graduation rate (growth 
trajectory) is included in the analysis, this study has three different hierarchical 
structures. Accordingly, three different types of sub-models will be generated. 
The first is the institutional growth model, which includes each individual 
institution’s growth trajectory for FTIC graduation rate. The second is the within- 
state model, which reflects institutional characteristics. The third is the between- 
states model, which analyzes the effects of state-level policy variables. 

The first step in model building is to identify the growth trajectory of each 
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institution between linear and polynomial models. To identify the growth 
trajectory, a visual inspection of all the institutions’ growth trajectories was 
conducted, and the average growth trajectory was generated and visually 
inspected as Bryk and Raudenbush (1992) recommend. Based on the identified 
growth trajectory, a linear within-institution model (Level-1 model) was 
generated. 

Prior to specifying institution-level and state-level models, it is useful to fit the 
unconditional model which does not include explanatory variables at the 
institution and state levels. The major purpose of the unconditional model is to 
collect information about the growth trajectory and point of origin (i.e., the 
estimated FTIC graduate rate in 1997). Based on the information obtained 
through unconditional models, the first step is to determine if sufficient variability 
among institution- and state-level variables exists. For example, if the growth 
rate within a given state is the same for every institution, it means that the 
parameter is constant across all the institutions within a state (Level-2). In this 
case, the parameter is retained at Level-2 and the corresponding random effect 
term is set to zero, but no Level-2 level predictors are included in the conditional 
model. 

Based on the results of the analysis of the unconditional model, the conditional 
model, which includes explanatory variables at Level-2 and Level-3, is 
considered. When the coefficients of the unconditional model are significant at 
each level, the variables are included in the conditional model. By introducing 
explanatory variables at each level, the total variation of FTIC graduation rate 
can be apportioned by levels. At Level-2, institution-level control variables are 
included, and state-level control variables and program variables of interest are 
included at Level-3. 

Results 

Results of the Unconditional Model 

Fitting the unconditional model at Level-2 and Level-3 exclusive of any Level-2 
or Level-3 predictors provided useful information about the general pattern of 
growth and the difference in the growth rates between institutions and states. 

Fixed Effects 

The state-level fixed effect of graduation rate at the beginning of the five-year 
period indicated that, averaged across all institutions in the 41 states included in 
the analysis, the estimated graduation rate in 1997 was 39.92 percent (t = 

28.23; p<. 001). The estimated state-level fixed effect of slope indicated that, 
averaged across all institutions in the 41 states, the growth in graduation rates 
was estimated to be 0.62 percentage points per year. This growth trajectory was 
significantly different from zero (t = 4.47; p<. 001). (See Table 1.) 

Random effect 
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The estimated variance components appear in the lower panel of Table 1 . At the 
institution level (Level-2), the large chi-square values (1545.11 and 585.84) 
indicate that there were large amounts of random variations (p<. 001) among 
institutions for both initial graduation rate and the growth in graduation rate. 
Therefore, further conditional analyses at the institution level were warranted. 
Across the states (Level-3), a significant amount of random variation also 
existed in the state mean initial graduation rate and state mean growth rates. 
The chi-square values for the two parameters were 132.72 (p<. 001) and 121 .10 
(p= <.001 ), respectively. Again, further conditional analysis for each parameter 
at the state level was also indicated. 


Table 1 

Fixed Effects and Random Effects (Unconditional Model) 


Fixed Effect 

Confident 

S.£. 

t-ratio 

P 

Average Initial Status, ;.ooo 

39.92 

1.41 

28.23 

<.001 

Average Grot\th Rate, ;/Joo 

0.62 

0.14 

4.47 

<.001 

Random Effect 

Variance 

df 

t: 

p 

Level 1 (within institutions) 





\'aiiation over time, etij 

926 




Les el 2 (between institution) 





Institution mean initial status, roij 

193.17 

314 

1545.11 

<.001 

Institution mean growth rate, rlij 

1.02 

314 

585.84 

<.001 

Level 3 (betw een states) 





State mean initial status, uooj 

52.07 

40 

132.72 

<.001 

State mean growth rate, uloj 

0.50 

40 

121.10 

<.001 


Model Specification at Institutional Level and Sate Level 

Based on the results of the unconditional model, the next step was to fit the 
institution-level model (Level-2 model) in order to account for random variation 
found in the Level-1 growth parameters. To search for Level-2 predictors with 
sufficient predictive power, all Level-3 models were temporarily left 
unconditional. Then, the Level-3 predictors were considered when the Level-2 
model was specified. 

Institution Level Model (Level-2 Model) 

To explain the variability between institutions, institutional characteristics were 
considered at Level-2. Among the three potential Level-2 variables, average 
tuition and financial student ratio were excluded in the analysis because both 
variables were highly correlated with other included variables. 

For the institution’s initial status, the variance component associated with 
student dormitory ratio (277.36) was relatively large, and also had statistically 
significant predictive power {t = 6.598; p< .001). Also, the residual variance for 
student dormitory ratio was significant (chi-square = 63.70, p = 0.008). For the 
institution’s growth rate, dormitory ratio was the only significant predictor (chi- 
square = 85.40, p < 0.001). 
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Therefore, student dormitory ratio was included for institution’s initial status and 
for institution’s growth rate. The final Level-2 conditional models were specified 
as: 


Institution’s mean initial status, 7ioii= pooj + /x)1j * {Dorm Ra{e)ij + roij 
Institution’s mean growth rate, 7tlii = p1oj + p11j* (Dorm Rate)// + r1ii 

State Level Model (Level-3 Model) 

To explain the differences between state variations, state characteristics were 
included in the models. The results of a correlation analysis suggest that family 
ability to pay college costs was the strongest potential predictor. (College 
preparation and unemployment rate were highly correlated with family ability to 
pay, and were thus excluded from the analysis because of multicollinearity.) 
Therefore, the final state-level models were: 

Mean initial status, pooj = yooo + }Oo1 (Family Ability) + uooi 
Dorm Rate, j3o1j= }o1o +:o11 (Family Ability) + uo1j 
Mean growth rate, /31oj = ytoo + ;/fof (Family Ability) + u1oj 
Dorm Rate, /311j = y11o (Family Ability) + u11j 

The Effects of PBF Programs 

The next step was to test the PBF program effects by including the program- 
related variables in the model. The first research question was to compare the 
growth in institutional graduation rates between PBF states and non-PBF states. 
To test for PBF program effects, the final model was expanded by adding a PBF 
program variable. This program variable was included only in the state average 
growth rate parameter - not the initial status parameter — because the research 
question concerned the effects of the PBF program on the growth in graduation 
rates. Therefore, the final models to test program effects were: 

Mean initial status, j6bo/ = ;ooo + ;ooo * (Family Ability)/ + uooj 

Dorm Rate, ;ofo + ,voff * (Family Ability)y + uo1j 

Mean growth rate. /31oj = /too + /o1 * (Family Ability)/ + /to2 * PBFy + u1oj 

Dorm Rate, = ;Tfo+ * (Family Ability)y+/712* PBF/ + u11j 


In the model, “PBF” represented a contrast between PBF states and non-PBF 
states. The effect of PBF programs on FTIC graduation rate was tested in two 
ways. First, the model fit improvement was tested to know whether significantly 
more state-level variance in graduation rate was explained by the introduction of 
PBF program variable. This test was conducted using a model fit test with D- 
statistics. Second, single parameter tests of the contrast of interest — PBF 


http:/ / epaa.asu.edu/ epaa/vl2n22/ 


5/25/2004 


EPAA Vol. 12 No. 22 Shin & Milton: The Effects of Performance Budgeting and Funding ... Page 13 of 27 


states vs. non-PBF states — were conducted. 

The model fit was not improved significantly by adding the PBF program 
variable (chi-square = 0.188; df= 2; p>.500). This result suggests that the PBF 
program variable did not help to reduce the unexplained variance at the state 
level. (See Table 2.) No matter whether the states were PBF or non-PBF states, 
the inclusion of the PBF program variable did not influence significantly the 
growth in graduation rates. In addition, the results for the single parameter tests 
showed that the contrast for PBF states vs. non-PBF states was not statistically 
significant (for state mean growth rate t = -0.50; p >.500, and for Dorm Rate t = 
0.044; p = >.500). (See Table 3.) 


Table 2 

Fixed Effects of Final Model (Testing PBF Program Effects) 

Fixed Effect Coefficient S£. t-ratio p 


For Initial status, joiJ 
For Average Initial Status, ^oj 
Intercept, ;«>oo 
Family Abilit>', ;ool 
For Dorm Rate, jBoJj 
Intercept, ;-oio 
Family Abilit>', ;^11 
For Gro\Mh Rate, ,tHJ 
For Average Growth rate, ^lOj 
Intercept, /loo 
Family Ability', viol 
PBF,;-'io2 
For Dorm Rate, ^IIJ 
Intercept, yj lo 
Family Abilit>', '{111 
PBF,/; 72 


39.64 

1.30 

31.01 

<.001 

0.54 

0.31 

1.77 

0.084 

27.82 

4.12 

6.76 

<.001 

2.73 

0.88 

3.11 

0.004 


0.76 

0.28 

2.69 

0.011 

-0.08 

0.03 

-2.30 

0.027 

-0.15 

0.31 

-0.50 

0.618 

-0.30 

1.48 

-0.18 

0.856 

-0.30 

0.14 

-2.16 

0.037 

0.07 

1.60 

0.04 

0.966 


Table 3 

Random Effects of Final Model (Testing PBF Program Effects) 
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RandomE ffect 

Variance 

df 

X* 

P 

Level 1 (within institutions) 

\'ariation over time, etij 
Level 2 (between institution) 

For institution initial status, noi] 

Institution mean initial status. 

9.25 

150.40 

273 

1399.8 

<.001 

roij 

For institution grot\th rate, nlij 

Institution mean gro\\th rate, rlij 

0.77 

273 

8 

483.28 

<.001 

Level 3 (between states) 

For institution mean initial status, yfloo/ 
State mean initial status, vooj 

43.39 

38 

105.45 

<.001 

State mean Dorm Rate, volj 

98.68 

38 

48.96 

0.110 

For institution mean growth rate, pioj 
State mean growth rate, uloj 

0.50 

37 

111.18 

<.001 

State mean Dorm Rate, ullj 

4.08 

37 

70.11 

0.001 


The Effects of Program Types 

Research question 2 concerned the effects of PBF types on growth in 
graduation rates within the PBF states. That is, did states with both PB and PF 
programs have higher growth in graduation rates than states with only a PB or 
PF program? Therefore, a new data file with only PBF states (30 states) was 
obtained by splitting the original data file. Fitting the model followed the same 
steps described above. 

The variance for the Level-2 model indicates that there were significant amounts 
of random variations (p<. 001) among institutions for both initial graduation rate 
and growth of graduation rate. Therefore, further conditional analysis at the 
institution level was warranted. Across states, too, a significant amount of 
random variation existed in the state mean initial status, and in the state mean 
growth rate. 

Based on the unconditional model, institutional characteristic variables were 
included at Level-2 models. Each predictor variable for an institution’s initial 
status and growth rate were the same as for the previous model specification. 
Through the model specification procedures, student dormitory ratio was 
selected for the institution’s initial status predictors and for the institution’s 
growth rate predictor. 

Based on the results of the Level-2 conditional model, state-level characteristics 
were included to construct a conditional Level-3 model. The model specification 
procedures were similar to those of the previous model, which was based on 41 
states. Family ability to pay for college proved significant and was included in 
the final model. In addition, the performance program type variable was included 
in the final model to address the research question. Therefore, the final models 
were: 
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/3ooi = yooo + uooj 

/3o1j= yo1o + :o11* (Family Ability)] + uo1j 
pioi- /loo + y1o1 * (Family Ability)/ + y1o2* (PBF 
P11j= y11o + y1 11* (Family Ability); + y1 12* (PBF 


Type]/ + lyfo; 
J\lpQ)i+ ullj 


By adding a PBF program type variable in the models, the model fit improved 
significantly (chi-square = 8.12; df=2,p = 0.017). In addition, single parameter 
test results showed that the PBF program type was significant overall. The 
effects of PBF program type on the state mean growth in graduation rates was 
statistically significant {t = 2.36; p = 0.026). Also, PBF program type had a 
significant effect through student dormitory ratio: in states with both PB and PF 
programs, the ratio of dormitory beds has a greater impact on graduation rate 
than in states with only one program {t = 2.61 ; p = 0.01 5) as shown in Table 4. 
The test results demonstrate that the states with both PB and PF programs 
performed better than the states with only a PB or PF program. 

Table 4 

Fixed Effects of Final Model (Testing PBF Program Types) 

Fixed Effect Coefficient S£. t-ratio p 


For Initial status, xoij 
For Average Initial Status, pooj 


Intercept, }ooo 

40.55 

1.49 

27.16 

<001 

For Dorm Rate, polj 

Intercept, /olo 

32.07 

5.15 

623 

<001 

Family Ability', ;-oii 

1.59 

1.04 

1.52 

0.139 

For Growth Rate, xlij 
For Average Growth rate, piOj 

Intercept, yioo 

0.40 

0.14 

2.75 

0.011 

Family Abilit\’, ylol 

-0.05 

0.03 

-1.73 

0.095 

PB & PF, }lo2 

0.55 

0J3 

2J6 

0.026 

For Dorm Rate, pi IJ 

Intercept, ;d lo 

-1.05 

0.69 

-1.52 

0.141 

Family Abilit>', yIII 

-0.27 

0.14 

-1.88 

0.071 

PB & PF, yil2 

3.18 

122 

2.61 

0.015 


Summary And Discussion 

The purpose of this study was to determine whether states with performance 
budgeting and funding (PBF) programs had increased the effectiveness of their 
public institutions of higher education over the five years considered in this 
study. To explore this question, two primary research questions were generated. 
An FILM analysis made it possible to test PBF program effectiveness: 1) 
between PBF states and non-PBF states; 2) between the states with both PB 
and PF programs and with only PB or PF program. The following is a summary 
of the research findings: 
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(1 ) During the five years under study, the growth of graduation rates in PBF 
states was not greater than in non-PBF states. 

(2) The growth in graduation rates in the states with both PB and PF programs 
was higher than in the states with only one (PB or PF) program. 

These results could be disappointing for state policy-makers who support 
performance-based reforms, or might be good news to those who disapprove of 
state government-initiated reforms in higher education. The following discussion 
explores the results and considers their implications. 

The first finding to consider is that the growth in graduation rate in PBF states 
was not found to be different from that in non-PBF states. In interpreting these 
results, it is important to consider: a) the nature of change in graduation rates at 
institutions of higher education; b) the amount of funding tied to PBF in a given 
state; and c) the degree to which PBF programs influence college- and 
departmental-level decision making. 

• As many higher education administrators have recognized, the change of 
the graduation rates among colleges and universities generally is very 
slow. This tendency is confirmed by a review of a decade of changes in 
graduation rates in Florida (Florida Board of Education, 2001), where the 
graduation rate at state colleges and universities varied by only four points 
from 1992 through 1999 (57.2 % in 1992, 58.7% in 1993, 59.5% in 1994, 
59.4% in 1995, 59.5% in 1996, 59.5% in 1997, 61.1 % in 1998, 59.6% in 
1999, and 59.9% in 2000). 

Despite changes in budgeting and other policies made by the state during this 
decade, the graduation rate of Florida institutions of higher education remained 
very stable, a fact that could be attributed to the intrinsic nature of change in 
higher education institutions and one which could have limited the impact of 
PBF programs on graduation rate. Such an alternative explanation for the slow 
growth of the graduation rate in PBF states may account for the lack of 
difference seen between PBF and non-PBF states, which would otherwise 
suggest that PBF programs are ineffective. 

The tendency of graduation rates to change slowly calls into question the use of 
FTIC graduation rate as a program indicator, particularly for performance 
funding (PF) programs, which tightly link budget with institutional performance. 
When the graduation rate is selected as a performance indicator for a PF 
program, policy-makers should not expect instant results. Thus, the findings of 
this study would suggest that PF states which use graduation rate as a 
performance indicator take into account the caveats described above. 

Another possible explanation on the ineffectiveness of the PBF programs might 
be the small amount of funding tied to institutional performance. Although some 
researchers (Floyt, 2001; Serban, 1998; MGT of America, INC., 1999) argued 
that the amount of money may not be crucial in PBF programs, the SHEEO 
survey results show that the proportion of money linked to performance is too 
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small — from 0.5% to 4% across all PBF states except South Carolina, the only 
state that allocates 100% of its higher education budget based on institutional 
performance (Christal, 1998). A considerable amount of program ineffectiveness 
might be explained by the low monetary stakes tied to institutional performance. 

In addition, although many states adopted PBF program, these states still 
allocate most portion of their budget based on traditional criteria. As Wildavsky 
(1988) argued, even when state governments adopt new budgeting policies, the 
tradition of incrementalism remains entrenched in many state agencies and 
state legislatures. As is well known, the main budget allocation criteria in higher 
education are traditionally the number of full time equivalent (FTE) students. 
Therefore, one possible scenario in budgeting allocation is that low-performing 
institutions with high student enrollment might receive more money than high- 
performing institutions with lower student enrollments, as Bridges (1999) found 
in his dissertation on the PF program in Colorado. 

From the point of program implementation, one critical issue related to 
ineffectiveness of the PBF programs is the level at which the PBF program has 
an impact within the institution. Expectations related to a PBF program might be 
communicated and tied to funding at the institutional level, departmental level, 
or faculty level. Considering that faculty plays a large part in the performance of 
higher education institutions, it is important that they be ‘on board’ in order to 
improve institutional performance. As Burke and his associates (2000) found, 
however, performance funding tends to become invisible on campus below the 
level of the vice- president. In some extreme cases, Poisel (1998) found that 
presidents of community colleges do not exactly understand the programs at the 
beginning of the PBF program implementation. 

The second finding to consider is the greater growth of the graduation rate in 
states with both PB and PF programs than in states with only one of these 
programs. To complement the strengths and accommodate the weaknesses of 
the two programs, ten states adopted both programs at the same time, thereby 
combining the flexibility of performance budgeting and the certainty of 
performance funding. 

The states with both programs might transfer strong political intentions to their 
colleges and universities by adopting the PF program; and they may allow for 
needed flexibility by adopting the PB program. Regardless of whether the PBF 
programs were mandated by state legislature, and whether performance 
indicators are prescribed or not, the flexibility and participation of institutional 
administrators and faculty in the design and implementation of PBF programs is 
crucial. Participants in the program implementation process tend to feel more 
responsible for the outcomes (Coulson-Clark, 1999). As Van Vaught (1994) 
argued, some of the failures of government-initiated changes in higher 
education may be attributed to the independent nature of the academic 
profession. This characteristic of higher education institutions ensures that 
government-initiated reforms in higher education systems will fail unless they 
are flexible in their implementation - and take faculty culture into account. 
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Limitations 

This study has two limitations in generalizing its findings. First, although the 
most common performance indicator, FTIC graduation rate, is used to compare 
institutional performance, some states do not use FTIC graduation rate as a 
performance indicator. However, there is no comprehensive study of 
performance indicators nationwide, except a State Higher Education Executive 
Officers (SHEEO) survey which found that 32 of 33 states in the U.S. include 
graduation rate as one of their performance indicators (Christal, 1998). 

Second, FTIC graduation rate is only one of the definitions of graduation rate. 
The report ‘Measuring Up 2000’ (NCPPHE, 2000) used two different terms in its 
calculation of completion. One approach was FTIC graduation rate and the 
other type involved the proportion of total completion to total enrollment. Only 
one of these terms, FTIC graduation rate was used for this study because most 
states use FTIC graduation rate as their graduation rate indicator. 

Conclusions 

In the 1990s, higher education institutions faced the pressure of externally- 
imposed reforms designed to link budget with institutional performance. Among 
these reforms were performance budgeting and performance funding programs. 
As traditionally autonomous institutions, however, colleges and universities were 
slow in responding to the demand for change posed by such programs. As the 
results of this study show, the implementation of PBF programs did not have the 
immediate or dramatic impact on higher education institutions that policy- 
makers may have expected. Institutional performance — FTIC graduation rate in 
this study — did not improve markedly after the adoption of PBF the programs by 
states. This outcome might be attributed to the tendency of higher education 
institutions to change slowly, to the use of graduation rate as a measure of 
institutional performance or effectiveness, or to problems intrinsic to the PBF 
programs themselves. 

The lack of growth in institutional graduation rates, however, does not mean that 
PBF programs failed to achieve their goals. More time may have been 
necessary for changes to become apparent, or changes might have appeared 
indirectly rather than directly. From a strategic point of view, then, legislatures 
might do well to encourage institutions to engage in actions which will lead to 
long-term change. Concluding that PBF programs are not useful based simply 
on changes in graduation rate over a short period, then, is not advised. 

Another issue comes to light when evaluating the effectiveness of PBF 
programs, that is, the common practice of state legislatures of allocating monies 
for higher education annually. As long as state governments measure 
institutional performance annually and allocate budgets based on this annual 
measurement, institutions will direct more money and effort towards short-term 
rather than long-term fundamental changes. The pressure of annual budgetary 
decisions is accentuated when states review and sometimes change their 
performance indicators annually. In such cases, institutions are not sure if 
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today’s performance indicators will be next year’s indicators. Under these 
circumstances, it is not surprising that institutions tend to focus more on short- 
term than long-term efforts to increase their institutional performance. 

In consideration of the findings of this study and the evaluation issues discussed 
above, policy-makers are advised to sustain PBF programs long enough until 
such programs bear their fruits or prove ineffective. In addition, a distinction 
between short-term and long-term performance indicators is essential. Long- 
term indicators should be considered in the budget allocation process after a 
reasonable time has passed, or weighted in such a way as to guide institutions 
to focus more on long-term than short-term changes. If policy-makers attend to 
such details, PBF programs may prove effective in fostering the sort of 
institutional change that benefits all involved with higher education. 

Notes 

1 . This study was done during 1 993 to 1 994, and the ten states included were 
Colorado, Florida, Illinois, Kentucky, New York, South Carolina, Tennessee, 
Texas, Virginia, and Wisconsin. 

2. Branch campuses will generally be excluded from the analysis, except in 
cases where adequate information is available. 

3. In 2001, the Integrated Postsecondary Education Data System (IPEDS) 
began to provide nationwide graduation rate data, which had been collected 
since 1991 using a student tracking system. 

4. In some cases, a low graduation rate reflects the initial intentions of students 
who ultimately wish to transfer to another institution, a scenario most frequently 
seen at community colleges (Whigham, 2000). 

5. We excluded states with two years PBF experience. This excluded nine 
states from the analysis. 

6. The “Overall score” of preparation is calculated weighted based on students’ 
high school completion, K- 12 course taking, and K-12 student achievement. 
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